On the Convergence of Stochastic Iterative Dynamic Programming Algorithms

Recent developments in the area of reinforcement learning have yielded a number of new algorithms for the prediction and control of Markovian environments These algorithms including the TD algo rithm of Sutton and the Q learning algorithm of Watkins can be motivated heuristically as approximations to dynamic program ming DP In this paper we provide a rigorous proof of convergence of these DP based learning algorithms by relating them to the powerful techniques of stochastic approximation theory via a new convergence theorem The theorem establishes a general class of convergent algo rithms to which both TD and Q learning belong An important component of many real world learning problems is the tem poral credit assignment problem the problem of assigning credit or blame to individual components of a temporally extended plan of action based on the success or failure of the plan as a whole To solve such a problem the learner must be equipped with the ability to assess the long term consequences of particular choices of action and must be willing to forego an immediate payo for the prospect of a longer term gain Moreover because most real world problems involving prediction of the future consequences of actions involve substantial uncertainty the learner must be prepared to make use of a proba bility calculus for assessing and comparing actions There has been increasing interest in the temporal credit assignment prob lem due principally to the development of learning algorithms based on the theory of dynamic programming DP Barto Sutton Watkins Wer bos Sutton s TD algorithm addressed the problem of learning to predict in a Markov environment utilizing a temporal di erence operator to update the predictions Watkins Q learning algorithm extended Sutton s work to control problems and also clari ed the ties to dynamic pro gramming In the current paper our concern is with the stochastic convergence of DP based learning algorithms Although Watkins and Watkins and Dayan proved that Q learning converges with probability one and Dayan observed that TD is a special case of Q learning and there fore also converges with probability one these proofs rely on a construction that is particular to Q learning and fail to reveal the ties of Q learning to the broad theory of stochastic approximation e g Wasan Our goal here is to provide a simpler proof of convergence for Q learning by making direct use of stochastic approximation theory We also show that our proof extends to TD for arbitrary Several other authors have recently pre sented results that are similar to those presented here Dayan and Sejnowski for TD Peng and Williams for TD and Tsitsiklis for Q learning Our results appear to be closest to those of Tsitsiklis We begin with a general overview of Markov decision problems and DP We introduce the Q learning algorithm as a stochastic form of DP We then present a proof of convergence for a general class of stochastic processes of which Q learning is a special case We then discuss TD and show that it is also a special case of our theorem Markov decision problems A useful mathematical model of temporal credit assignment problems studied in stochastic control theory Aoki and operations research Ross is the Markov decision problem Markov decision problems are built on the formalism of controlled Markov chains Let S N be a discrete state space and let U i be the discrete set of actions available to the learner when the chain is in state i The probability of making a transition from state i to state j is given by pij u where u U i The learner de nes a policy which is a function from states to actions Associated with every policy is a Markov chain de ned by the state transition probabilities pij i There is an instantaneous cost ci u associated with each state i and action u where ci u is a random variable with expected value ci u We also de ne a value function V i which is the expected sum of discounted future costs given that the system begins in state i and follows policy